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EXECUTIVE  SUMMARY 


Continuous  adult  learning  is  the  hallmark  of  establishing  and  maintaining  compe¬ 
tence  and  expertise  in  the  military  occupations.  This  learning  environment  relies  increas¬ 
ingly  on  technology  to  be  continuously  accessible  to  learners  and  other  users.  Because  of 
the  autonomous  nature  of  technology,  such  learning  environments  will  place  on  the  devel¬ 
opers  an  enormous  responsibility  to  design  and  implement  materials  from  the  learner’s  per¬ 
spective. 

Technology-based  systems  for  education,  training,  and  performance-aiding  (which 
includes  decision-aiding)  may  pose  the  ultimate  test  for  validating  approaches  to  integrate 
humans  and  automated  systems.  These  systems  need  to  model  students  and  users.  The 
models  they  generate  and  the  interactions  based  on  these  models  must  adapt  to  the  evolving 
knowledge  and  skills  of  individual  students  and  users.  Evaluation  findings  suggest  that  such 
adaptations  are  feasible,  worthwhile,  and  cost-effective.  Data  drawn  from  many  evaluations 
of  technology-based  education  and  training  indicate  that  these  systems  can  reduce  costs  by 
one-third  and  that  they  can  also  either  reduce  the  time  to  achieve  instructional  objectives  by 
one-third  or  increase  achievement  (holding  time  constant)  by  one-third.  The  likely  impact  on 
military  readiness  and  effectiveness  suggested  by  these  findings  is  significant. 

Evaluations  of  technology-based  performance-aiding  systems  have  resulted  in 
similar  findings  regarding  increased  personnel  effectiveness  and  cost-savings.  These 
evaluations  suggest  a  need  to  review  and  readjust  the  balance  between  resources  allocated  to 
training  and  resources  allocated  to  performance-aiding  systems.  The  development  of 
sharable,  reusable  objects  for  the  generation  of  instructional  (or  decision-aiding)  presenta¬ 
tions  and  the  capabilities  for  assembling  these  objects  on  demand  and  in  real  time  will 
substantially  increase  accessibility  and  reduce  costs  of  education,  training,  and  performance- 
aiding  while  making  them  asynchronously  and  continuously  available  regardless  of  distance 
and  time. 

Specifications  and  capabilities  for  such  objects  are  the  goals  of  much  research  and 
development  (R&D).  Some  of  these  goals  are  discussed  under  the  systems  engineering 
categories  of  analysis,  design  and  development,  delivery  and  management,  and  evaluation. 
North  Atlantic  Treaty  Organization/Partnership  for  Peace  (NATO/PfP)  participation  would 
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substantially  enhance  this  research  agenda.  This  participation  might  include  the  development 
of  a  NATO/PfP  directory  of  databases  that  would  permit  wide  dissemination  and  sharing  of 
techniques  and  findings  and  the  development  of  NATO/PfP  common  practices  for 
developing  sharable  objects  and  guidelines  for  then-  use. 
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COMPUTER-BASED  AIDS  FOR  LEARNING, 
JOB  PERFORMANCE,  AND  DECISION-MAKING 
IN  MILITARY  APPLICATIONS: 
EMERGENT  TECHNOLOGY  AND  CHALLENGES 


A.  INTRODUCTION 

Continuous  adult  learning  is  the  hallmark  of  establishing  and  maintaining  compe¬ 
tence  and  expertise  in  the  military  occupations.  This  learning  environment  relies  increas¬ 
ingly  on  technology  to  be  continuously  accessible  to  learners  and  other  users.  Because  of 
the  autonomous  nature  of  technology,  such  learning  environments  will  place  on  the  devel¬ 
opers  an  enormous  responsibility  to  design  and  implement  materials  from  the  learner’s  per¬ 
spective.  Education,  training,  and  performance- aiding  systems  may  be  the  ultimate  test  bed 
for  validating  technological  approaches  to  accommodate  humans  in  automated  systems. 

Automated  instructional  systems  must  address  several  variables,  including  the 
method  of  instructional  delivery,  desired  learning  objectives,  and  learning  environment  (see 
Figure  1).  Figure  1  illustrates  these  elements  in  three  dimensions: 

•  The  “Complexity  of  Technology  &  Technique”  axis  provides  a  continuum  of 
technical  difficulty  beginning  with  computer-aided  instruction  (CAI)  and  pro¬ 
gressing  in  difficulty  to  a  virtual  reality  (VR)  distributed  simulation.  Educa¬ 
tional  objectives  and  the  availability  of  required  equipment  determine  the  level 
of  technology  required  for  instruction. 

•  The  “Fevel  of  Feaming”  axis  concerns  the  level  of  knowledge  required  by  the 
task(s)  to  be  performed.  Benjamin  Bloom’s  taxonomy  offers  a  convenient 
framework  for  defining  levels  of  learning— from  basic  knowledge  to  the  more 
involved  process  of  evaluation  (Bloom,  1956).  These  levels  suggest  the 
instructional  techniques  to  be  used  and  affect  the  training  time  needed  to  reach 
them. 

•  The  “Environment”  axis  denotes  where  the  information  learned  will  be  used, 
ranging  from  a  familiar  simple  environment  (e.g.,  the  office)  to  the  unfamiliar, 
complex  environments  characteristic  of  military  operations  (e.g.,  the  battlefield). 
The  environment  determines  what  training  devices  will  be  available  and  helps 
set  priorities  for  instructional  objectives. 
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Figure  1.  Dimensionality  of  Human  Roles  in  Training  Systems 

A  key  point  emphasized  in  this  document  is  that  each  human  will  play  many  roles  in 
his/her  interactions  with  any  system,  including  (as  Figure  1  suggests)  training  systems.  This 
figure  suggests  that  these  roles  must  evolve  along  with  the  level  of  learning  achieved  by 
individuals.  Capable  systems  (notionally,  the  numbered  items  in  Figure  1)  must 
accommodate  these  levels  of  learning  dynamically  as  users  achieve  them— adjusting  the 
complexity  of  technologies,  the  techniques  they  employ,  and  the  environment  they  present  to 
the  learners.  To  be  effective  in  providing  continuous  adult  learning,  both  the  system  and  the 
users  must  change  their  behavior.  Both  must  adapt  and  learn. 

Can  we  expect  this  to  happen?  Will  technology,  specifically  computer  technology, 
adapt  to  its  users  in  much  the  same  way  that  users  today  must  adapt  to  it?  The  technology  is 
certainly  becoming  more  powerful.  After  almost  40  years,  Moore’s  law  (e.g.,  Service,  1996) 
still  holds.  In  1965,  Gordon  Moore1  noted  that  engineers  were  doubling  the  number  of 
electronic  devices  (basically  transistors)  on  chips  every  year.  In  1975,  he  revised  his 


1  Gordon  E.  Moore  is  currently  Chairman  Emeritus  of  Intel  Corporation.  Moore  co-founded  Intel  in 
1968,  serving  initially  as  Executive  Vice  President.  He  became  President  and  Chief  Executive  Officer 
in  1975  and  held  that  post  until  elected  Chairman  and  Chief  Executive  Officer  in  1979.  He  remained 
CEO  until  1987  and  was  named  Chairman  Emeritus  in  1997. 
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statement  to  say  that  this  doubling  was  occurring  every  2  years.  If  we  split  the  difference 
and  predict  it  will  occur  every  18  months,  our  expectations  fit  reality  quite  closely. 

One  consequence  of  Moore’s  law  is  that  significant  computational  power  and  func¬ 
tional  capacity  are  found  in  today’s  desktops  and  laptops  for  less  than  $1,000  per  system. 
Moore,  however,  did  not  predict  the  pace  of  progress  in  human-centered  design  of  computer 
software  interfaces  or  in  software  applications.  Raymond  Kurzweil  discussed  this  rapid 
increase  in  electronic  capability  [10  years  ago  in  the  The  Age  of  Intelligent  Machines  (1992) 
and,  more  recently,  in  The  Age  of  Spiritual  Machines  (2000)].  Kurzweil  predicted  that  a 
$1,000  unit  of  computing  would  equal  the  computational  capability  of  the  human  brain  by 
the  year  2019— and  exceed  it  thereafter.  Whether  this  $1,000  unit  will  be  “intelligent”  and 
have  functional  capability  adaptively  centered  on  the  human  user  can  be  debated,  but  the 
rapid  expansion  of  computational  capability  into  many  areas  of  human  cognition  seems 
undeniable. 

The  pervasive  nature  of  automation  technology  and  applications  to  military  enter¬ 
prises  seems  equally  undeniable.  It  is  present  not  only  in  our  offices,  schools,  and  homes, 
but  it  has  also  become  an  essential  enabling  capability  for  our  military  operations.  We  seem 
inexorably  borne  on  a  rising  crest  of  digital  applications  in  command  and  control  (C2), 
communications,  modeling  and  simulation  (M&S),  sensors  and  surveillance,  and,  notably,  in 
such  human-intensive  activity  as  decision-making,  education,  training,  and  performance- 
aiding.  These  latter  applications  and  their  inherent  technology  challenges  are  the  topic  of 
this  document,  which  addresses  the  following  questions: 

•  Is  it  worth  it?  What  do  we  know  about  the  value  of  technology  applications  in 
learning  (education  and  training)  and  performance-aiding  (decision-making  and 
job-task  enhancement)  and  what  are  the  challenges  if  we  want  to  realize  the  full 
value? 

•  Where  do  we  want  to  go?  What  are  promising  technological  opportunities 
and  directions  for  these  applications  and  how  might  we  best  employ  them  to 
support  military  operations  and  activities? 

•  How  should  we  get  from  here  to  there?  How  can  science  and  engineering 
advance  the  state  of  the  art  and  the  state  of  our  practice  in  performing  military 
operations? 

B.  IS  IT  WORTH  IT? 

We  will  argue  “yes.”  Scientists  and  engineers  have  been  pursuing  research  and 
development  (R&D)  on  and  assessments  of  computer  applications  in  education,  training, 
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and  performance- aiding  for  more  than  45  years  (Fletcher  and  Rockway,  1986).  Much, 
possibly  most,  of  this  work  in  the  United  States  has  been  funded  by  the  military  (Congress 
of  the  United  States,  Office  of  Technology  Assessment,  1988)  and  is  intended  to  enhance 
military  capabilities.  This  pattern  of  funding  support  may  also  be  true  for  other  countries 
(Chatelier  and  Seidel,  1993;  Seidel  and  Chatelier,  1995).  It  is  not  surprising,  then,  to  find 
that  data  on  the  effectiveness  and  cost  benefits  of  these  applications  have  been  accumulating. 
We  might  wish  for  more  data,  particularly  in  the  area  of  cost  benefits;  however,  the  weight 
of  evidence  makes  it  much  easier  to  argue  (on  the  basis  of  existing,  empirical  data)  for  the 
value  of  these  applications  rather  than  against  them.  These  data  are  reviewed  in  the  following 
sections  on  education  and  training  applications,  human  instructors  in  the  loop,  and 
performance- aiding  applications. 

1.  Education  and  Training  Applications 

Suppose  we  could  reduce  by  30  percent  the  time  that  military  personnel  need  to 
reach  a  given  level  of  training.  We  would  then  expect  to  see  reductions  in  expenditures  for 
instructor  pay  and  allowances,  student  pay  and  allowances,  temporary  duty  costs,  training 
equipment  costs,  and  installation  support  costs  — among  the  measurable  items.  Not  sur¬ 
prisingly,  we  find  that  savings  from  the  reduction  of  training  time  can  be  substantial— a 
“value”  that  may  go  beyond  the  measurable  (e.g.,  flexibility  in  the  reuse  of  the  recovered  or 
“saved”  time). 

For  instance,  the  U.S.  military  spends  about  $4  billion  a  year  on  specialized  skill 
training.  This  is  the  training  required  after  “basic”  or  accession  training  to  qualify 
personnel  for  the  many  technical  jobs  (e.g.,  wheeled  vehicle  mechanics,  radar  operators, 
avionics  technicians,  medical  technicians)  needed  to  conduct  military  operations.  It  does  not 
include  the  cost  of  aircraft  pilot  training,  which  comprises  a  separate  cost  category.  Figure  2 
shows  the  annual  reductions  in  costs  that  would  result  if  instructional  time  were  reduced  by 
30  percent  for  20,  40,  60,  and  80  percent  of  the  U.S.  military  personnel  who  complete 
specialized  skill  training  each  year.  For  instance,  if  the  United  States  military  were  to  reduce 
by  30  percent  the  time  to  train  20  percent  of  the  personnel  undergoing  specialized  skill 
training,  it  would  save  over  $250  million  per  year.  If  it  were  to  do  so  for  60  percent  of  the 
personnel  undergoing  specialized  skill  training,  it  would  save  over  $750  million  per  year,  an 
appreciable  amount  by  almost  any  standard. 
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Figure  2.  Monetary  Savings  in  Specialized  Skill  Training 
With  a  30-Percent  Reduction  in  Training  Time 

Time  savings  of  30  percent  are  used  here  for  a  reason.  As  Table  1  suggests,  savings 
of  about  this  magnitude  in  training  time  are  frequently  found  in  reviews  of  instructional 
technology.  Orlansky  and  String  (1977)  reported  that  reductions  in  time  to  reach 
instructional  objectives  averaged  about  54  percent  in  their  review  of  technology  used  in 
military  training.  Fletcher  (1991)  found  an  average  time  reduction  of  31  percent  in 
6  assessments  of  interactive  videodisc  (IVD)  instruction  applied  in  higher  education.  Kulik 
(1994)  reported  time  reductions  of  34  percent  in  17  assessments  of  technology  used  in 
higher  education  and  24  percent  in  15  assessments  of  adult  education.  All  these  reviews 
were  independent  in  that  they  reviewed  different  sets  of  evaluation  studies.  On  this  basis,  it 
does  not  seem  unreasonable  to  expect  instructional  technology  applications  to  reduce  the 
time  it  takes  students  to  reach  a  variety  of  given  instructional  objectives  in  military  education 
and  training  by  about  30  percent. 


Table  1.  Percent  Time  Savings  for  Technology-Based  Instruction 


Study  (Reference) 

Number  of  Studies 

Reviewed 

Average  Time  Saved 
(Percent) 

Orlansky  and  String  (1977) 

13 

54 

Fletcher  (1991) 

8 

31 

Kulik  (1994)  (Higher  Education) 

17 

34 

Kulik  (1994)  (Adult  Education) 

15 

24 
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For  that  matter,  30  percent  is  a  conservative  target.  Commercial  enterprises  that 
develop  technology-based  instruction  for  the  Department  of  Defense  (DoD)  regularly  base 
their  bids  on  the  expectation  that  they  can  reduce  instructional  time  by  50  percent  (Fletcher, 
personal  communication).  Noja  (1991)  has  reported  time  savings  as  high  as  80  percent 
through  the  use  of  technology-based  instruction  in  training  operators  and  maintenance  tech¬ 
nicians  for  the  Italian  Air  Force. 

a.  Where  Do  These  Time  Savings  Come  From? 

To  a  large  extent,  these  time  savings  may  be  accounted  for  by  individualization  of 
pace— the  speed  with  which  students  move  through  instructional  material  and  reach  instruc¬ 
tional  objectives.  Even  the  most  rudimentary  of  technology-based  instruction  systems  adjust 
pace  for  individual  students. 

b.  Does  the  Capability  To  Adjust  Pace  Matter? 

Many  classroom  instructors  have  been  struck  by  the  differences  in  the  pace  at  which 
their  students  learn,  and  research  confirms  their  observations.  For  instance,  consider  the 
following  findings  on  the  time  that  different  students  take  to  reach  the  same  instructional 
objectives: 

•  Ratio  of  time  needed  by  individual  hearing-impaired  and  Native  American  stu¬ 
dents  to  reach  mathematics  objectives:  4  to  1  (Suppes,  Fletcher,  and  Zanotti, 
1975;  1976) 

•  Overall  ratio  of  time  needed  by  individual  students  to  learn  in  grades  K-8:  5  to 
1  (Gettinger,  1984) 

•  Ratio  of  time  needed  by  undergraduates  in  a  major  research  university  to  learn 
features  of  the  LISt  Processor  (LISP)2  programming  language:  7  to  1  (Private 
communication,  Corbett,  1998) 

We  may  not  be  surprised  to  discover  differences  among  students  in  the  speed  with 
which  they  are  prepared  to  leam,  but  the  magnitudes  of  the  differences  do  seem  surprising. 
As  we  might  expect  from  Gettinger’s  1984  review,  a  typical  Kindergarten  through  Grade  8 
classroom  will  have  students  who  are  prepared  to  learn  in  one  day  what  it  will  take  other 
students  in  the  same  classroom  5  days  to  leam.  This  difference  does  not  seem  to  be  miti¬ 
gated  by  more  homogeneous  grouping  of  students  based  on  their  abilities.  The  students  in 


-  LISP  is  generally  regarded  as  the  language  for  artificial  intelligence  (AI).  LISP  was  formulated  by  AI 
pioneer  John  McCarthy  in  the  late  1950s. 
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Corbett’s  (1998)  research  university  are  highly  selected,  averaging  well  above  the  80th  per¬ 
centile  on  their  admission  tests,  yet  the  differences  in  time  they  require  to  learn  a  modestly 
exotic  programming  language  remain  large. 

The  differences  in  the  speed  with  which  different  students  reach  given  objectives 
may  be  caused,  initially,  by  ability,  but  this  effect  is  quickly  overtaken  by  prior  knowledge  as 
a  determinant  of  pace  (Tobias,  1989).  This  is  likely  to  be  particularly  true  of  students  in 
military  education  and  training  who  bring  a  wide  variety  of  backgrounds  and  life  experi¬ 
ences  to  the  classroom. 

The  challenge  this  diversity  presents  to  classroom  instructors  is  daunting.  How  can 
they  ensure  that  every  student  has  enough  time  to  reach  given  instructional  objectives?  At 
the  same  time,  how  can  they  assist  those  students  who  are  ready  to  surge  ahead?  The 
answer,  of  course,  despite  heroic  efforts  to  the  contrary,  is  that  they  cannot.  Most  class¬ 
rooms  contain  many  students  who,  at  one  end  of  the  spectrum  are  bored  and,  at  the  other 
end,  are  overwhelmed  and  lost.  Technology  allows  us  to  alleviate  this  difficulty  by  adjusting 
the  pace  of  instruction  to  the  needs  and  abilities  of  individual  students.  They  can  proceed  as 
rapidly  or  as  slowly  as  needed.  They  can  easily  skip  what  they  already  know  or  have 
mastered  and  concentrate  on  what  they  have  yet  to  learn. 

c.  Do  These  Savings  in  Time  Come  at  the  Expense  of  Instructional 
Effectiveness? 

The  data  suggest  the  opposite  (i.e.,  the  use  of  technology  decreases  instruction  time 
and  increases  instruction  effectiveness).  Noja’s  1987  findings  are  not  uncommon.  For  the 
Italian  Air  Force  technicians,  in  comparing  conventional  instruction  in  electronics  with 
technology-based  instruction,  he  found  the  following: 

•  A  reduction  in  training  time  of  3  weeks  (from  8  weeks  to  5  weeks) 

•  Equivalent  student  achievement  for  electronic  theory 

•  Substantial  improvements  in  student  achievement  for  electronic  applications. 

One  study  does  not  provide  final  answers,  but  many  studies  can  be  aggregated  to 
suggest  conclusions.  This  aggregation  is  usually  done  using  “meta-analysis”  (analysis  of 
analyses),  with  an  estimation  of  effect  sizes.  Roughly,  effect  sizes  are  normalized  measures 
of  standard  deviations  found  by  subtracting  the  mean  from  one  collection  of  results  (e.g.,  a 
control  group)  from  the  mean  of  another  (e.g.,  an  experimental  group)  and  dividing  the 
resulting  difference  by  an  estimate  of  their  common  standard  deviation.  Because  effective 
sizes  are  normalized,  they  can  be  averaged  to  give  an  overall  estimate  of  effect  from  many 
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separate  studies  undertaken  to  investigate  the  same  phenomenon  or  treatment.  Figure  3 
shows  effect  sizes  from  several  collections  of  studies  that  compared  conventional  instruction 
with  three  types  of  technology-based  instruction. 


Figure  3.  Some  Effect  Sizes  for  Studies  Comparing  Technology-Based 
Instruction  With  More  Conventional  Approaches 

In  Figure  3,  “Computer-Based  Instruction”  (CBI)  summarizes  results  from 
233  studies.  These  studies  involved  straightforward  application  of  computer  presentations 
that  used  text,  graphics,  and  some  animation— as  well  as  some  degree  of  individualized 
interaction.  The  effect  size  of  0.39  standard  deviations  suggests,  roughly,  that  the  perform¬ 
ance  levels  of  50th  percentile  students  improved  to  the  performance  levels  of  65th  percentile 
students. 

“Interactive  Multimedia  Instruction”  involves  more  elaborate  interactions  adding 
more  audio,  more  extensive  animation,  and  (especially)  video  clips.  Greater  achieve¬ 
ment— an  average  effect  size  of  0.50  standard  deviations  compared  with  an  effect  size  of 
0.39  for  typical  CBI— may  compensate  for  the  added  cost  of  these  capabilities.  An  effect 
size  of  0.50  for  interactive  multimedia  instruction  suggests  that  the  performance  levels  of 
50th  percentile  students  improved  to  the  performance  levels  of  69th  percentile  students. 

“Intelligent  Tutoring  Systems”  (ITSs)  involve  a  capability  that  has  been  developing 
since  the  late  1960s  (Carbonell,  1970)  but  has  only  recently  been  expanding  into  general 
use.  In  this  approach,  an  attempt  is  made  to  mimic  directly  the  one-on-one  dialogue  that 
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occurs  in  tutorial  interactions.  The  key  component  is  that  computer  presentations  and 
responses  are  generated  in  real-time,  on  demand  and  as  needed  or  as  requested  by  learners. 
Mixed-initiative  dialogue  in  which  either  the  computer  or  learner  can  ask  or  answer  open- 
ended  questions  is  supported.  These  interactions  are  generated  as  required.  Instructional 
designers  do  not  need  to  anticipate  and  pre-store  them.  This  approach  is  computationally 
more  sophisticated,  and  it  is  more  expensive  to  produce  than  standard  CBI.  However,  its 
costs  may  be  justified  by  the  increase  in  average  effect  size  to  0.84  standard  deviations, 
which  suggests,  roughly,  that  the  performance  levels  of  50th  percentile  students  improved  to 
the  performance  levels  of  80th  percentile  students. 

Some  later  ITSs  (Gott,  Kane,  and  Lesgold,  1995)  were  considered  just  to  see  how 
this  approach  is  progressing.  The  average  effect  size  of  1.05  standard  deviations  for  these 
recent  applications  is  promising.  It  represents,  roughly,  that  the  performance  levels  of  50th 
percentile  students  to  the  performance  levels  of  85th  percentile  students. 

The  more  extensive  tailoring  of  instruction  to  the  needs  of  individual  students  that 
can  be  obtained  using  generative  ITSs  can  only  be  expected  to  increase.  Such  systems  may 
raise  the  bar  for  the  ultimate  effectiveness  of  technology -based  instruction.  They  may  make 
available  far  greater  efficiencies  than  we  can  now  obtain  from  other  approaches,  but  they  will 
require  instructional  and  computational  capabilities  that  today  are  only  partially  in  hand. 
They  will  need  a  full  range  of  subject  matter  expertise  and  sufficient  insight  to  present  it  to 
students.  A  mathematical  model  of  a  system  will  help,  but  it  will  not  suffice.  These  genera¬ 
tive  ITSs  will  have  to  “know”  what  they  themselves  know  and  what  the  user  does  not.  They 
will  need  an  unobtrusive  assessment  capability  that  allows  them  to  model  dynamically  what 
students  have  learned,  have  not  yet  learned,  and  have  misunderstood.  Very  probably,  these 
systems  will  need  to  communicate  with  their  human  users  through  natural  (human) 
language,  which  implies  not  just  speech-fragment  recognition,  but  also  comprehensive  lan¬ 
guage  and  discourse  understanding.  We  will  need  the  intelligent,  if  not  spiritual  machines, 
Kurzweil  has  said  that  we  can  expect  in  17  years  or  so. 

2.  Human  Instructors  in  the  Loop 

In  the  interim,  and  doubtless  after  that,  we  will  need  human  instructors  in  the  loop. 
How  best  to  use  these  human  instructors  is  a  perennial  issue  in  the  design  and  implemen¬ 
tation  of  technology  based  instruction.  Finding  the  right  balance  is  important.  In  preparing 
technicians  of  the  Italian  Air  Force,  Noja  (1991)  has  attributed  much  of  his  success  to 
achieving  a  reasonable  balance  of  effort  between  technology  and  human  instructors. 
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In  the  United  States,  as  in  perhaps  many  countries,  the  loss  of  military  experts  (e.g., 
through  retirement,  vocational  burnout,  and  similar  factors)  in  critical  areas  has  combined 
with  drastic  reductions  in  live  training  experiences  (because  of  costs,  range,  and 
environmental  restrictions,  greater  reach  of  weapons,  and  so  forth)  to  increase  the  need  to 
make  the  best  use  of  remaining  military  expertise.  One  significant  way  to  do  this  is  to  sup¬ 
port  experts  who  are  performing  instruction,  namely,  by  providing  technology-based  dis¬ 
plays  that  are  intimately  linked  to  the  subject  matter  being  presented  and  the  enabling 
knowledge  and  skills  that  students  need  to  master  it.  The  Interactive  Multi-sensor  Analysis 
Training  (IMAT)  system  developed  by  the  U.S.  Navy  to  support  training  for  antisubmarine 
warfare  sensor  operators  is  an  example  of  such  a  capability. 

IMAT  provides  training  in  sensor  deployment,  adversary  detection,  and  undersea 
warfare  tactical  battlespace  “sensemaking”  (e.g.,  environmental  analysis,  sensor  selection 
and  placement,  search  rate  and  threat  detection,  multi-sensor  crew  coordination,  and  multi¬ 
sensor  information  integration)  to  compensate  for  the  diminishing  number  of  opportunities 
to  develop  and  sustain  these  skills  in  formal  training,  on-the-job  training  (OJT),  and  fleet 
exercises.  IMAT  integrates  models  of  the  physical  phenomena  and  innovative  visualization 
techniques  to  demonstrate  relationships  among  threats  (submarines  and  their  weaponry),  the 
undersea  environment,  and  anti-submarine  systems.  It  combines  analytic  and  instructional 
design  technologies  and  advanced  computer-based  graphics  to  promote  rapid  acquisition  of 
the  cognitive  visualization  capabilities  sailors  need  to  understand  structural  and  spatial 
interrelationships  among  sensors,  platforms,  and  submarine  systems. 

The  combination  of  instructor- led,  IMAT- supported  training  has  not  only  raised 
performance  against  established  instructional  objectives,  but  it  has  also  raised  the  level  of 
objectives  set  for  training.  It  helped  sailors  achieve  levels  of  competence  that  simply  were 
not  available  without  it.  Notably,  IMAT  fulfills  most  of  the  training  capabilities  suggested 
by  Figure  1.  It  allows  training  to  be  ported  from  basic  to  complex  operational  environments. 
It  allows  training  environments  to  use  human  instructors  and  advanced  technology  together 
to  achieve  more  high-level  (transferable,  abstract,  problem-solving)  objectives  than  were 
attainable  using  traditional  training  approaches,  and  it  does  so  at  easily  affordable  costs.  It 
supports  training  in  familiar,  relatively  simple  environments  while  being  scaleable  to 
environments  characteristic  of  “live”  military  operations.  In  short,  IMAT  training  allows 
students  to  accomplish  far  more  than  either  instructors  or  technology,  working  separately, 
can  achieve. 
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3.  Performance-Aiding  Applications 

Problem-solving  is  required  when  an  individual  or  a  group  must  achieve  a  goal  but 
is  uncertain  how  to  do  so  (Baker  and  Mayer,  1999;  Mayer  and  Wittrock,  1996).  It  requires 
ingenuity  and  creativity  to  manipulate  and  transform  the  knowledge  and  skills  that  problem- 
solvers  possess  into  paths  of  action  leading  to  the  goal.  It  is  a  necessary  and  integral  com¬ 
ponent  of  human  performance  in  all  sectors,  and  it  is  a  critical  component  in  every  military 
operation.  Difficulty  in  making  problem-solving  decisions  is  exacerbated  by  the  frequency 
with  which  we  are  confronted  with  too  much  data,  too  many  options,  and  unknown  levels  of 
risk.  These  matters  have  been  the  object  of  systematic  study  by  both  past  (James, 
1890/1950)  and  present  (Edwards  and  Fasolo,  2001)  researchers. 

Given  the  complexity  of  real-world  decision-making  and  the  range,  both  descriptive 
and  prescriptive,  of  its  theoretic  underpinnings,  seeking  assistance  from  technology  does  not 
seem  unreasonable.  Such  applications  involve  tightly  integrated  interactions  between 
humans  and  automated  systems.  They  raise  some  of  the  most  interesting  issues  for  R&D  in 
determining  the  role  of  humans  in  the  use  of  intelligent  and  automated  systems. 

Not  enough  studies  of  the  effectiveness  and  cost-benefits  of  these  applications  exist 
to  permit  a  meta-analytic  assessment  of  their  impact.  However,  systematic  and  compre¬ 
hensive  assessment  of  technology-based  performance-aiding  can  be  found  in  the  collection 
of  studies  covering  the  Integrated  Maintenance  Information  System  (IMIS),  an  automated, 
hand-held,  flight-line  avionics  maintenance  aid.  Tomasetti  et  al.  (1993)  documented  a  thor¬ 
ough  cost  analysis  of  IMIS.  Thomas  later  (1995)  reported  results  from  an  empirical  inves¬ 
tigation  of  IMIS  effectiveness.  Teitelbaum  and  Orlansky  (1996)  summarized  results  from 
these  studies,  combined  them  into  a  more  complete  cost-effectiveness  assessment,  and  dis¬ 
cussed  the  implications  of  these  findings. 

Thomas  (1995)  compared  the  performance  of  12  Avionics  Specialists  and  12  Air¬ 
plane  General  (APG)  Technicians  on  12  fault-isolation  problems  concerning  3  F-16  avion¬ 
ics  subsystems:  fire  control  radar,  heads-up  display,  and  inertial  navigation.  Within  each  of 
the  two  groups  of  subjects,  six  of  the  fault-isolation  problems  were  performed  using  paper- 
based  task  orders  (TOs)  (from  Air  Force  technical  manuals),  and  six  were  performed  using 
IMIS.  Training  for  APG  Technicians  includes  all  aspects  of  aircraft  maintenance,  only  a 
small  portion  of  which  concerns  avionics.  In  contrast,  Avionics  Specialists  receive  16  weeks 
of  specialized  training  in  avionics  maintenance.  Table  2  shows  the  results  of  the  study. 
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Table  2.  Maintenance  Performance  of  12  Air  Force  Avionics  Specialists  and 
12APG  Technicians  Using  TOs  and  IMIS 


Technicians/ 

Performers 

Correct 

Solutions 

(Percent) 

Time  to 

Solution 

(Minutes) 

Average 
Number  of 

Parts  Used 

Time  to 

Order  Parts 
(Minutes) 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

Avionics  Specialists 

81.9 

100.0 

149.3 

123.6 

8.7 

19.4 

1.2 

APG  Technicians 

69.4 

98.6 

175.8 

124.0 

8.3 

5.3 

25.3 

1.5 

Observations  that  might  be  made  from  these  results  include  the  following: 

•  Avionics  Specialists  using  TOs  compared  with  those  using  IMIS.  The 

Avionics  Specialists  using  IMIS  found  more  correct  solutions  in  less  time, 
used  fewer  parts  to  do  so,  and  took  less  time  to  order  these  parts.  All  these 
results  were  statistically  significant.  The  number  of  parts  required  may  deserve 
brief  comment.  Savings  in  spare-parts  inventory  and  transportation  were  by  far 
the  largest  factors  in  the  Tomasetti,  et  al.  (1993)  analysis  of  costs.  The  parts 
reduction  exerted  considerable  leverage  on  the  overall  cost  savings  reported  by 
Teitlebaum  and  Orlansky  (1996).  The  reduction  in  time  to  order  parts  is  to  be 
expected  because  IMIS  automates  much  of  this  process.  Notably,  the  time 
taken  by  technicians  to  complete  the  paperwork  in  the  absence  of  IMIS  could 
be  used  elsewhere,  with  substantial  productivity  gains  and  cost  savings  if  IMIS, 
or  a  similar  capability,  performs  these  paperwork  chores. 

•  APG  Technicians  using  TOs  compared  with  those  using  IMIS.  Thomas 
(1995)  found  similar  results  in  these  comparisons.  APG  Technicians  using 
IMIS  performed  more  correct  solutions  in  less  time,  used  fewer  parts  to  do  so, 
and  took  less  time  to  order  these  parts.  As  with  Avionics  Specialists,  all  these 
results  were  statistically  significant. 

•  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using 
TOs.  APG  Technicians  using  IMIS  found  more  correct  solutions  in  less  time, 
used  fewer  parts  to  do  so,  and  took  less  time  to  order  these  parts  than  did 
Avionics  Specialists  using  paper-based  TOs.  All  these  results  were  statistically 
significant.  This  result  suggests  that  it  is  feasible  and  desirable  to  replace  some 
of  the  extra  training  required  by  specialists  with  on-the-job,  just-in-time 
decision  aids,  such  as  IMIS,  supplied  to  nonspecialists. 

•  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using 
IMIS.  In  these  comparisons,  APG  Technicians  performed  about  as  well  as 
Avionics  Specialists  and  even  slightly  better  in  the  number  of  parts  used.  None 
of  these  comparisons  were  statistically  significant,  and  none  appear  to  be 
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practically  significant.  These  results  again  suggest  the  feasibility  of  replacing 
some  number  of  specialists,  who  require  greater  training  costs,  with  general 
technicians  supplied  with  on-the-job,  just-in-time  decision  aids.  They  also 
suggest  the  desirability  of  doing  so  because  of  the  greater  costs  to  train  the 
specialists— even  though  the  resulting  performance  on  the  job,  where  it  counts, 
is  the  same  in  both  cases. 

The  promise  suggested  by  these  results  could  well  vanish  if  the  costs  to  provide  the 
decision  aid  (IMIS)  exceed  the  costs  they  otherwise  save.  However,  using  two  sources  of 
data  [i.e.,  the  costs  and  benefits  analysis  by  Tomasetti  et  al.  (1993)  combined  with  the 
empirical  results  reported  by  Thomas  (1995)],  Teitelbaum  and  Orlansky  (1996)  were  able  to 
estimate  reductions  in  depot-level  maintenance,  organizational-level  maintenance,  and  the 
maintenance  and  transportation  of  inventories  of  spare  parts.  Teitelbaum  and  Orlansky 
estimated  that  annual  savings  from  the  use  of  IMIS  would  be  approximately  $38  million  for 
the  full  Air  Force  fleet  of  about  1,700  F-16s.  Their  analysis  also  considered  the  costs  to 
develop  and  maintain  IMIS.  Assuming  an  8-year  useful  life  for  IMIS,  they  arrived  at  a  fig¬ 
ure  of  about  $18  million  per  year  to  maintain  IMIS  (including  its  databases)  and  to  amortize 
its  development  costs.  The  result  is  about  $20  million  per  year  in  net  savings. 

This  figure  of  $20  million  is  conservative.  Focusing  only  on  cost  may  underestimate 
the  total  “value”  of  this  technology.  It  does  not  include 

•  Savings  that  would  result  from  a  reduction  in  Air  Force  requirements  to  recruit 
and  train  specialized  personnel  such  as  the  Avionics  Specialists  in  Thomas’s 
study 

•  Savings  in  training  that  would  accrue  from  the  use  of  IMIS  as  a  decision  aid 
and  as  a  training  device 

•  Savings  in  the  costs  to  print,  distribute,  and,  especially,  update  paper  technical 
manuals 

•  Savings  (of  about  50  percent)  in  time  to  debrief  pilots  about  maintenance 
problems. 

Most  important,  these  benefits  do  not  include  those  arising  from  increased  sortie  rates  and 
enhanced  operational  readiness  and  effectiveness  resulting  from  the  substantially  improved 
problem-solving  competencies  of  maintenance  personnel. 

As  with  training  systems,  performance-aiding  will  benefit  from  what  appears  to  be 
the  inexorable  march  of  technology.  However,  significant  technology  challenges  remain. 
Performance-aiding  systems  will  require  interactive,  advising,  decision-aiding,  and  compu¬ 
tational  capabilities  that  today  are  only  partially  in  hand.  Just  like  training  systems,  they  will 
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need  a  full  range  of  subject  matter  expertise  and  sufficient  insight  to  present  it  to  users. 
They  will  have  to  “know”  what  they  themselves  know  and  what  the  individual  user  does 
not  know  so  they  provide  assistance  in  ways  that  the  user  is  prepared  to  understand  and  act 
on.  To  do  this,  they  will  need  an  unobtrusive  assessment  capability  that  allows  them  to 
model  dynamically  what  users  know,  do  not  know,  and  misunderstand.  Very  probably,  they 
will  need  to  communicate  with  their  human  users  using  natural  (human)  language  involving 
comprehensive  language  and  discourse  understanding.  They  too  will  need  the  “intelligent” 
capabilities  for  interacting  with  humans  that  is  foreseen  by  Kurzwiel  (and  others)  and  that 
researchers  developing  ITSs  are  creating.  These  systems  are  discussed  in  the  next  section. 

4.  Is  It  Worth  It?  A  Summary 

Given  the  weight  of  data  presented  here  and  elsewhere,  accelerating  the  research, 
development,  and  implementation  of  computer  technology  in  military  education,  training, 
and  performance-aiding  seems  reasonable.  However,  resources  for  such  a  pursuit  are  finite 
and,  most  likely,  cannot  support  all  that  remains  to  be  done.  Thus,  some  consideration  of 
technology  opportunities  and  priorities  driven  by  operational  requirements  seems  in  order. 
We  should  decide  where  we  want  to  go  before  trying  to  get  there. 

C.  WHERE  DO  WE  WANT  TO  GO? 

One  prominent  and  promising  vision  for  the  future  is  captured  by  the  notion  of 
asynchronous,  continuous  learning— which  is  education,  training,  and  performance-aiding 
that  is  available  anytime,  anywhere  to  whomever  needs  it.  Such  a  vision  capitalizes  on  the 
development  of  the  World  Wide  Web  (www)  (or  whatever  the  global  information  grid  of 
the  future  will  be),  generative  CBI,  unobtrusive  assessment  of  users’  intentions  and  needs, 
ITSs,  natural  language  understanding,  the  development  of  educational  objects,  and,  of 
course,  the  continuing  advancements  in  computation  and  computer  technology.  Figure  4 
captures  this  vision  roughly. 

This  vision  anticipates  a  future  in  which  everyone  will  have  access  to  an  electronic 
personal  learning  associate.  This  device  will  assemble  learning  or  decision-aiding  presenta¬ 
tions  on  demand  and  in  real-time— any  time,  anywhere.  The  presentations  will  be  tailored 
exactly  to  the  needs,  capabilities,  intentions,  and  learning  state  of  each  individual  or  group 
(e.g.,  crew,  team,  or  staff).  Communication  with  the  device  will  be  based  on  natural 
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Shareable  courseware  Assembled  in  To  provide  learning 

objects  from  across  the  real-time ,  on-  and  assistance 

World  Wide  Web  demand  anytime,  anywhere 


Figure  4.  Asynchronous,  Continuous  Learning:  A  Vision  for  the  Future 

language  dialogue  initiated  either  by  the  device  or  by  its  users.  The  device  will  small  enough 
to  be  carried  in  a  shirt  pocket,  or  it  will  be  wearable.  It  will  be  used  by  individuals  learning 
by  themselves,  in  groups,  or  in  classrooms.  It  will,  of  course,  be  wireless. 

Most  of  the  technology  needed  to  build  such  a  device  exists  now.  Although  we 
cannot  yet  fit  it  into  a  shirt  pocket,  creative  engineering  should  take  care  of  that.  What  is 
especially  needed  for  instruction  and  decision-aiding  is  content  in  the  form  of  instmctional 
objects,  which  we  are  calling  shareable  instructional  objects.  These  objects,  shown  in  the 
cloud  on  the  left  side  of  Figure  4,  must  be  readily  accessible  across  the  World  Wide  Web 
or  whatever  form  our  global  information  network  takes  in  the  future. 

Once  these  objects  exist,  they  must  be  identified,  selected,  and  assembled  in  real 
time,  on  demand  and  then  handed  to  the  personal  learning  associates,  which  provide  the 
instruction  or  decision-aiding.  This  work  of  identifying,  selecting,  and  assembling  objects  is 
the  job  of  the  server,  the  box  in  the  middle  of  Figure  4.  By  importing  “logic”  or  instruc¬ 
tional  strategy  objects,  the  server,  which  today  might  be  called  a  learning  management 
system,  can  acquire  the  capabilities  of  an  intelligent  tutoring  system.  This  vision  is  keyed  to 
the  development  and  implementation  of  intelligent  instructional  systems.  These  systems 
must  be  scaleable  and  adaptive  to  military  personnel  “occupational  specialties,”  the  variety 
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of  instruction  venues,  and  the  operational  domain(s)  addressed  by  the  application  (see  the 
mapped  with  notional  numbered  examples  in  Figure  5). 


Figure  5.  Dimensionality  of  Instructional  Systems 

Figure  5  represents  the  challenge  to  designer  and  developers  of  military  instructional 
systems  along  three  dimensions: 

•  The  “Student”  axis  concerns  the  type  of  student,  coupled  with  the  roles  and 
responsibilities  of  the  student.  These  roles  and  responsibilities  suggest  targeted 
levels  of  expertise,  abstraction,  and  objectives.  This  axis  points  to  the  need  for 
dynamic  adaptability  of  the  instructional  system. 

•  The  “Instructional  Venue”  axis  reflects  where  the  instruction  will  occur  (e.g., 
in  the  classroom,  the  work  environment,  duty  station,  garrison,  home,  or  some 
combination  of  these).  This  axis  points  to  the  need  for  curriculum  design  and 
the  instructional  infrastructure  flexibility  and  its  required  characteristics. 

•  The  “Student  Location”  axis  identifies  where  the  knowledge  gained  will  be 
used  (e.g.,  in  locations  ranging  from  garrison  and  depots,  to  field  combat 
command  centers,  to  the  battlefield).  It  determines  the  application  environment 
for  the  knowledge  and  skills  gained  and  their  criticality. 

Figure  5  suggests  the  kinds  of  intelligence  that  an  automated  instructional  system 
must  possess  to  operate  effectively  on  behalf  of  its  human  user.  It  also  suggests  dimensions 
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for  validating  instructional  and  performance-aiding  systems  to  ensure  that  the  instruction 
and  information  they  convey  can  be  trusted.  R&D  along  these  dimensions  will  directly 
impact  what  are  commonly  called  intelligent  tutoring  systems  or  ITSs. 

1.  Intelligent  Tutoring  Systems  (ITSs) 

It  may  be  best  to  note  the  features  that  “ordinary”  CBI  provides.  It  can 

•  Accommodate  an  individual  student’s  rate  of  progress,  allowing  as  much  or  as 
little  time  as  each  student  needs  to  reach  instructional  objectives 

•  Adjust  the  sequence  of  instructional  content  to  each  student’s  needs 

•  Adjust  the  content  itself  so  that  different  students  can  receive  different  content, 
depending  on  what  they  have  mastered  and  what  they  have  yet  to  learn 

•  Make  the  instruction  as  easy  or  as  difficult  as  necessary 

•  Adjust  to  the  learning  style  (e.g.,  verbal  vs.  visual)  that  is  most  appropriate  for 
each  student. 

These  capabilities  have  been  available  and  used  in  CBI  from  its  inception  in  the  1950s 
(Fletcher  and  Rockway,  1986).  Those  who  promote  systems  with  these  features,  touting 
them  as  indicators  of  newly  developed  “intelligent”  capabilities,  may  be  missing  some  his¬ 
tory.  They  are  using  the  term  “intelligent”  in  ways  that  differ  from  the  historical  objectives 
of  ITS— objectives  that  have  been  pursued  since  the  late  1960s. 

What  are  these  objectives?  What  is  left  for  ITS  to  provide?  What  can  we  get  from 
them  that  is  not  otherwise  available?  Two  functionalities  deserve  mention: 

1.  The  ability  to  allow  either  the  computer  or  the  student  to  ask  open-ended 
questions  and  initiate  instructional,  “mixed-initiative”  dialogue  as  needed  or 
desired 

2.  The  ability  to  generate  instructional  material  and  interactions  on  demand  rather 
than  require  developers  to  foresee  and  pre- store  all  the  materials  and 
interactions  needed  to  meet  all  possible  eventualities. 

The  first  functionality  requires  the  ITS  to  understand  and  participate  in  mixed- 
initiative  interactions  with  the  student.  It  requires  mutual  understanding  of  a  language  for 
information  retrieval,  decision-aiding,  and  instruction  that  is  shared  by  the  ITS  and  the 
student/user.  Natural  language  has  been  a  frequent  choice  for  this  capability,  but  the 
language  of  mathematics,  mathematical  logic,  and  electronics  have  also  been  used  (e.g., 
Suppes,  1981;  Sleeman  and  Brown,  1982;  Psotka,  Massey,  and  Mutter,  1988;  and  Farr  and 
Psotka,  1992).  Whatever  form  mixed-initiative  dialogue  takes,  a  key  feature  of  one-on-one 
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tutorial  instruction  is  dialogue  in  which  either  the  student  or  the  instructor  can  initiate 
interactions  (e.g.,  Graesser  and  Person,  1994).  Such  a  capability  has  been  a  long-standing 
goal  of  intelligent  tutoring  systems  (Carbonell,  1970). 

The  second  functionality  requires  ITSs  to  devise  on  demand— not  retrieve  from 
storage— interactions  and  presentations  for  individual  students.  This  capability  involves 
more  than  generating  elements  to  fill  in  blanks  in  a  template.  It  means  generating 
interactions  and  presentations  from  information  primitives  using  an  “instructional 
grammar”  that  is  analogous  to  the  deep  structure  grammar  of  the  transformational- 
generative  linguists  of  a  generation  ago.  This  functionality  harkens  back  to  the  roots  of  ITS 
development  as  (again)  can  be  seen  in  the  volumes  edited  by  Suppes  (1981),  Sleeman  and 
Brown  (1982),  Psotka,  Massey,  and  Mutter  (1988),  and  Farr  and  Psotka  (1992). 

Motivations  for  both  these  functionalities  can  be  found  in  basic  research  into  human 
learning,  memory,  perception,  and  cognition.  Findings  from  this  research  have  led  us  to  view 
all  cognitive  processes  as  constructive  and  regenerative.  They  have  caused  general  theories 
of  perception  and  learning 

•  To  evolve  from  the  fairly  strict  logical  positivism  of  behavioral  psychology 
(which  emphasized  the  study  of  directly  observable  and  directly  measurable 
actions)  to  a  greater  consideration  of  internal,  less  observable  processes  that  are 
assumed  to  mediate  and  enable  human  learning 

•  To  produce  the  directly  observable  behavior  that  is  the  subject  of  behaviorist 
approaches. 

The  keynote  of  these  conceptions  of  cognition  may  have  been  struck  by  Ulric  Neisser,  who 
stated,  “The  central  assertion  is  that  seeing,  hearing,  and  remembering  are  all  acts  of  con¬ 
struction,  which  may  make  more  or  less  use  of  stimulus  information  depending  on  circum¬ 
stances”  (1967,  p.  10).  However,  these  ideas  have  been  part  of  the  fabric  of  scientific 
psychology  since  its  inception  (e.g.,  James,  1890/1950). 

This  point  of  view  suggests  that  the  generative  capability  sought  by  the  vision  pre¬ 
sented  by  Figure  4  is  not  something  merely  nice  to  have,  but  is  also  essential  if  we  are  to 
advance  beyond  the  constraints  of  the  prescribed  branching,  programmed  learning,  and 
ad-hoc  principles  currently  used  to  design  technology-based  instruction.  A  generative 
approach  may  be  essential  if  we  are  to  deal  successfully  with  the  immensity,  extent,  and 
variability  of  human  cognition. 

The  key  defining  characteristic  of  ITSs  then  is  not  the  application  of  computer  tech¬ 
niques  from  artificial  intelligence  (AI)  or  knowledge  representation  or  the  specification  of 
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sharable  instructional  objects,  important  as  these  may  be.  Rather,  it  is  the  functional  capa¬ 
bility  to  generate  in  real  time  and  on  demand  instructional  interactions  that  are  tailored  to 
student  requests  and/or  needs.  This  generative  capability  motivated  DoD  to  invest  in  the 
development  of  these  systems  in  the  first  place  (Fletcher  and  Rockway,  1986).  At  that  time, 
the  motivation  was  to  reduce  or  eliminate  the  high  costs  of  foreseeing  or  predicting  all  pos¬ 
sibly  needed  materials  and  interactions,  programming  them,  and  pre- storing  them  into  CBI. 
Today,  this  motivation  remains  fundamental  to  R&D  in  intelligent  systems  intended  to 
augment  the  learning  and  performance  of  human  beings. 

2.  An  Example  of  an  ITS  at  Work 

An  example  of  these  capabilities  at  work  can  be  found  in  the  student-computer, 
mixed- initiative  dialogues  supported  by  the  Sophisticated  Instructional  Environment 
(SOPHIE),  an  early  and  seminal  ITS  (Brown,  Burton,  and  DeKleer,  1982)  that  would  fit  in 
Figure  5  at  the  lower  front  left.  SOPHIE  used  a  general-purpose  electronic  simulator  to 
represent  an  electronic  power  supply  and  create  a  reactive,  tutorial  environment  for  training 
in  electronic  troubleshooting.  The  underlying  simulator  allowed  faults  introduced  into  the 
system  by  an  (human)  instructor  or  by  SOPHIE  to  be  propagated  throughout  the 
(simulated)  power  supply  in  a  realistic  fashion.  SOPHIE  employed  a  semantic  network  of 
information  about  the  power  supply,  one  of  the  most  sophisticated  natural  language 
understanders  ever  built  into  an  ITS,  and  an  articulate,  expert  troubleshooting  capability  that 
could  not  only  solve  arbitrary  faults  introduced  by  students  or  instructors,  but  also  explain 
to  the  student  how  it  solved  them. 

SOPHIE’s  reactive  environment  responded  in  a  realistic  fashion  to  the  actions  and 
decisions  of  a  student  engaged  in  practical  problem-solving.  It  employed  strategies  that 
shadowed  those  used  by  the  student  to  make  inferences  about  the  circuit  based  on  the 
student’s  measurements.  SOPHIE  could  thereby  determine  which  deductions  the  student 
was  using,  construct  a  model  of  his/her  state  of  knowledge,  and  use  this  model  to  generate 
explanations  in  terms  that  the  student  was  prepared  to  understand.  SOPHIE  highlighted  the 
need  for  ITSs  not  only  to  solve  arbitrary  problems,  but  also  to  do  so  in  a  way  that  can  be 
explained  to  students  and  be  understood  by  them.  It  pioneered  capabilities  currently  lacking 
in  many  information  acquisition  and  retrieval  systems. 

The  following  is  a  student-computer  dialogue  adapted  from  SOPHIE: 

Student:  What  is  the  base  emitter  voltage  of  the  voltage-limiting  transistor? 

SOPHIE:  The  base  emitter  voltage  of  Q5  is  -.037  volts. 
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The  first  student-SOPHIE  interaction  in  this  example  is  a  straightforward  question 
and  answer.  SOPHIE  has  introduced  a  fault  into  the  simulated  power  supply  and  the  stu¬ 
dent  is  trying  to  locate  and  repair  it. 

Student:  What  about  for  the  current  limiting  transistor? 

SOPHIE:  The  base  emitter  voltage  of  Q8  is  .349  volts. 

The  second  student-SOPHIE  interaction  is  a  little  less  straightforward  because,  in 
addition  to  simulating  the  faulted  power  supply,  SOPHIE  must  be  able  to  parse  the  dialogue 
sufficiently  to  know  that  the  student  is  asking  for  the  base  emitter  voltage  of  transistor  Q8. 
The  meaning  of  the  student’s  question  depends  heavily  on  the  context  established  by 
his/her  prior  question. 

Student:  What  should  it  be? 

SOPHIE:  In  a  working  circuit,  the  base  emitter  voltage  of  Q8  is  .593  volts. 

The  third  student-SOPHIE  interaction  continues  its  dependence  on  SOPHIE’ s 
understanding  of  context  and,  in  addition,  requires  SOPHIE  to  extract  the  answer  by 
simulating  both  the  faulted  power  supply  and  one  that  is  operating  correctly. 

Student:  Replace  Q8. 

SOPHIE:  I  am  going  to  ask  you  some  questions  about  how  Q8  is  faulted.  Are 
any  junctions  shorted? 

The  fourth  student-SOPHIE  interaction  indicates  a  clear  step  beyond  what  Brown, 
Burton,  and  DeKleer  (1982)  described  as  a  knowledgeable  system  to  what  they  considered 
to  be  an  intelligent  system.  SOPHIE  has  progressed  from  a  knowledgeable  parsing  of  its 
dialogue  with  the  student  and  simulation  of  various  states  of  the  power  supply  to  a  system 
exercising  tutorial  intelligence.  It  shadowed  the  student’s  solution  path,  modeled  the  stu¬ 
dent’s  troubleshooting  hypotheses,  determined  that  they  are  incorrect,  elected  to  capture  the 
dialogue  initiative  back  from  the  student,  and  is  undertaking  a  series  of  tutorial  interactions 
intended  to  lead  the  student  back  to  a  more  correct  approach  to  the  problem. 

It  is  difficult  to  imagine  any  practical  way  to  achieve  this  level  of  functionality  with¬ 
out  the  generative  capability  and  mixed-initiative  dialogue  that  distinguishes  ITSs  from  other 
forms  of  CBI.  Such  functionality  is  as  applicable  and  essential  to  performance- aiding  as  it 
is  to  instruction. 
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3.  Where  Do  We  Want  To  Go?  A  Summary 


To  this  point,  we  have  described  a  vision  of  CBI  in  a  general  training  or  education 
application,  which,  in  a  military  context,  is  the  foundation  for  developing  domain  expertise  in 
military  specialties.  Can  this  vision  be  applied  to  the  adult  learning,  occupational  training 
domain  that  is  the  foundation  of  military  effectiveness? 

The  answer  should  be  “yes,”  but  that  “yes,”  as  suggested  earlier  in  this  document, 
is  limited  by  the  instructional,  performance-aiding,  and  computational  capabilities  available 
today.  We  have  speech  recognition,  but  the  levels  of  language  and  discourse  understanding 
we  need  have  to  be  developed.  We  have  valid  models  of  military  systems,  but  we  lack  the 
“executive  functions”  that  will  allow  automated  instructional  and  performance- aiding  sys¬ 
tems  to  map  the  expertise  represented  by  these  models  onto  human  users  and  match  their 
presentations  to  individual  user’s  needs.  We  have  authoring  systems,  but  they  need  addi¬ 
tional  development  to  reduce  to  an  acceptable  level  the  costs  and  time  required  to  produce 
instructional  and  performance- aiding  systems.  Above  all,  we  lack  a  comprehensive  set  of 
empirically  based  design  principles  to  support  an  engineering  of  instructional  and  perform¬ 
ance-aiding  systems  that  would  lead  reliably  to  the  achievement  of  given  instructional  out¬ 
comes  and  substantially  reduce  the  time  to  produce  human  expertise  and  competence,  or 
even,  as  IMAT  does,  produce  levels  of  expertise  in  military  personnel  that  currently  do  not 
exist.  The  next  section  addresses  what  is  needed  to  produce  such  capabilities. 

In  brief,  the  vision  depicted  in  Figure  4  and  applied  in  the  dimensions  portrayed  by 
Figure  1  and  Figure  5  implies  several  key  goals: 

•  The  product  of  this  R&D  must  be  capable  of  generating  presentations  on 
demand,  in  real  time. 

•  Content  analysis  (e.g.,  cognitive  task  analysis)  must  be  able  to  be  conducted  by 
trainers  and  educators  ubiquitously. 

•  Its  presentations  must  be  tailored  in  content,  sequence,  level  of  difficulty,  level 
of  abstraction,  style,  and  so  forth  to  users’  intentions,  backgrounds,  and  needs. 
Neither  students  nor  problem- solvers  will  benefit  from  a  device  that  does  not 
provide  information  they  are  capable  of  understanding.  Efficiency  in  learning 
and  performance  also  demands  such  tailoring  to  user  needs  and  capabilities. 

•  It  must  be  used  equally  well  to  aid  learning  or  performance. 

•  It  must  accommodate  mixed-initiative  dialogue  in  which  either  the  technology 
or  the  user  can  initiate  or  respond  to  inquiries  in  natural  language. 
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•  It  will  benefit  greatly  from  a  supply  of  sharable  instructional  objects  readily 
available  for  the  generation  of  instructional  (or  decision-aiding)  presentations. 

These  goals  suggest  requirements  for  R&D— requirements  that  must  be  met  if  we  are  to  get 
from  here  (now)  to  there  (where  we  want  to  be). 

D.  HOW  SHOULD  WE  GET  FROM  HERE  TO  THERE? 

1.  Overview 

Much  has  to  be  done  before  we  will  achieve  the  vision  presented  in  this  document 
for  education,  training,  and  performance-aiding.  However,  accomplishing  some  R&D  goals 
will  result  in  products  that  can  be  used  immediately,  and  these  goals  might  be  achievable  in 
the  short  term. 

A  DoD  workshop  on  continuous  learning  systems  held  in  October  1999  laid  the 
foundation  for  these  recommendations.  This  workshop  was  motivated  in  large  part  by  the 
recognition  that  military  readiness  and  effectiveness  are  keyed  not  just  to  the  supply  of 
systems  and  personnel,  which  are  commonly  measured  in  U.S.  assessments  of  readiness, 
but  also  to  the  cognitive  capabilities  of  military  personnel— to  what  might  be  described  as 
their  cognitive  readiness  (Etter,  Foster,  and  Steele,  2000). 

Table  3  is  a  summary  and  an  elaboration  of  findings  from  this  workshop.  It  is 
roughly  organized  into  the  standard  system  engineering  categories  of  analysis,  design,  and 
development;  delivery  and  management;  and  evaluation. 

Table  3  lists  and  briefly  describes  potential  elements  of  a  research  agenda  for  the 
development  of  technology-based  aids  for  education,  training,  performance- aiding,  and 
decision-making  in  military  applications.  These  elements  do  not  comprise  a  formally  estab¬ 
lished  research  agenda,  but  they  suggest  the  breadth  of  R&D  that  must  be  undertaken  to 
capitalize  on  the  promise  offered  by  our  new  technologies  for  education,  training,  and  per¬ 
formance-aiding.  The  elements  suggest  opportunities  for  cooperative,  international  devel¬ 
opment  undertaken  by  North  Atlantic  Treaty  Organization/Partnership  for  Peace  (NATO/ 
PfP)  countries. 
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Table  3.  Capabilities,  Goals,  and  R&D  Requirements  for  the  Development  of 
Technology  in  Education,  Training,  and  Performance-Aiding 


Capability 

Description/Goal 

Requirements/Gaps 

Analysis 

Instruction 

Analysis: 

Cognitive  theory 

Accurate  and  comprehensive 
representation  of  cognitive  skill 
and  cognitive  skill  development  to 
include  such  “higher  order” 
capabilities  as  decision-making, 
problem-solving,  meta-cognition, 
pattern  recognition,  critical 
thinking,  situational  awareness, 
and  teamwork. 

New  concepts  and  theories  of 
cognition  based  on  new 
measurement  capabilities, 
especially  those  available  from 
technology-based  assessment; 
representation  of  expertise  in 
complex,  ill-structured 
environments. 

Instruction 

Analysis: 

Training  analysis 

Efficient,  comprehensive,  and 
accurate  analysis  of  military  needs 
for  command,  job,  and  task 
performance. 

Automation  of  processes  for 
performing  front-end  analysis, 
cognitive  analysis,  and  cognitive 
readiness  assessment; 
comprehensive  and  accurate 
representation  of  individual  and 
crew,  team,  and  unit  expertise, 
capabilities,  and  performance; 
principles  for  determining  relevant 
task  characteristics  for 
instructional  design  and  selection 
of  instructional  strategies. 

Instruction 

Analysis: 

Integrated  human 

performance 

system 

Training  analysis,  design,  and 
selection  of  objectives  integrated 
with  the  full  range  of  alternatives 
used  to  ensure  human 
competence  in  military  operations. 

Comprehensive  principles  for 
optimal,  cost-effective,  and 
balanced  adjustment  of  tradeoffs 
between  all  elements  (e.g., 
selection,  personnel 
classification,  training,  ergonomic 
design,  job  design)  used  to 
ensure  the  presence  of  human 
competence  when  and  where  it  is 
needed. 

Instruction 

Analysis: 

Technology-based 

knowledge 

representation 

Verified  and  validated 
representations  of  expert 
solutions,  capabilities,  and 
behavior. 

Automated  capture  of  expertise; 
self-generating,  self-modifying 
databases  built  from  cases  and 
examples  of  successful  problem¬ 
solving  and  decision-making. 

Instruction 

Analysis: 

Assessment  of 
learner  potential 

Ability  to  use  computer 
technology  to  develop  more 
comprehensive  and  accurate 
models  of  human  cognition  and 
potential;  ability  to  identify 
potential  “aces”  in  specific  militarily 
relevant  areas. 

New  forms  of  computer- 
administered  assessment  items 
using  the  full  display  and  timing 
capabilities  of  technology. 
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Table  3.  Capabilities,  Goals,  and  R&D  Requirements  for  the  Development  of 
Technology  in  Education,  Training,  and  Performance-Aiding  (Continued) 


Capability 

Description/Goal 

Requirements/Gaps 

Design  and  Development 

Instruction  Design 
and  Development: 

Instructional 

objects 

Wide  availability  of  accessible, 
portable,  reusable  instructional 
components  or  “objects”;  fully 
developed  infrastructure  to 
support  their  use. 

Object-oriented  programming  for 
instruction,  models,  and 
simulations;  standards  for 
developing  sharable  instructional 
and  simulation  objects. 

Instruction  Design 
and  Development: 

Management  of 
student  progress 

Comprehensive  engineering  of 
instruction  that  links  instructional 
design  and  presentations  to 
targeted  instructional  outcomes 
(e.g.,  speed  of  response, 
accuracy  of  response,  retention  of 
knowledge  and  skills,  motivation 
to  continue  learning). 

Principles  of  design  and 
presentation  that  can  be 
automated  as  needed  to  ensure 
reliable  achievement  of 
instructional  outcomes  by 
individuals,  crews,  teams,  and 
units. 

Instruction  Design 
and  Development: 

Cognitive 

readiness 

development 

Development  of  comprehensive 
techniques  for  measuring  and 
enhancing  cognitive  readiness. 

Principles  for  the  development  of 
higher  level  cognitive  skills  (e.g., 
creativity,  intuition,  pattern 
recognition,  transfer  of  know¬ 
ledge  and  skills,  problem-solving, 
and  situational  awareness). 

Instruction  Design 
and  Development: 

Human-behavior 

representation 

Verified  and  valid  representation 
of  individual  and  group  behavior 
for  virtual  team  members, 
adversaries,  friendly  forces, 
noncombatants,  and 
organizational  behavior  for  C2, 
and  nongovernmental 
organizations. 

Principles  for  developing 
physically  and  cognitively  realistic 
avatars;  principles  for  matching 
the  level  of  realism  in  human 
behavior  and  depiction  to  training 
objectives;  accurate  and  facile 
interchange  of  real  and  virtual 
participants;  development  of 
sharable,  reusable  human 
behavior  objects;  capabilities  that 
allow  dynamic  control  of  avatars  by 
either  human  instructors  or 
automated  tutors. 

Instruction  Design 
and  Development: 

Representation  of 
knowledge  and  skill 

Automated  representation  of  both 
subject  matter  expertise  and 
misconceptions. 

Comprehensive  computer 
representation  of  human 
knowledge  and  skill;  capabilities 
for  automated  development, 
verification,  and  validation  of 
these  representations. 

Instruction  Design 
and  Development: 

Articulate  expertise 

Ability  to  inform  students  and  aid 
problem-solvers  in  terms  they  are 
prepared  to  understand. 

Ability  to  assemble  objects  and 
prepare  presentations  that  are 
tailored  to  dynamic 
representations  of  the  student  or 
problem-solver. 
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Table  3.  Capabilities,  Goals,  and  R&D  Requirements  for  the  Development  of 
Technology  in  Education,  Training,  and  Performance-Aiding  (Continued) 


Capability 

Description/Goal 

Requirements/Gaps 

Instruction  Design 
and  Development: 

Natural  language 
understanding 

Ability  to  engage  in  natural 
language  dialogues  with  students 
and  problem-solvers. 

Speech  recognition;  ability  to 
parse  natural  language;  ability  to 
parse  dialogues;  adequate 
grammar  for  natural  language  and 
natural  language  dialogue. 

Instruction  Design 
and  Development: 

Authoring  tools 

Fully  developed  capability  for 
automated  development  of 
instructional  and  performance- 
aiding  materials  and  optimally 
efficient  tools  for  human  use  when 
automation  is  not  practicable. 

Comprehensive  capability  for 
generating  instruction  and 
performance-aiding  materials  on 
demand,  in  real  time,  tailored  to 
the  specific  needs  of  individuals, 
crews,  teams,  and  units;  ability  to 
generate  instruction  and 
performance-aiding  automatically 
from  computer-aided  design  and 
computer-aided  manufacturing 
databases. 

Delivery  and  Management 

Instruction  Delivery 
and  Management: 

Assessment  of 

learner/user 

knowledge 

Ability  to  assess  automatically  and 
unobtrusively  student/user 
knowledge,  skills,  and  needs  from 
routine  interactions  with  software. 

Computer  representation  of  user 
knowledge  skills,  and  needs; 
generation  of  valid,  unobtrusive 
near-real-time  assessment  from 
interactions  of  individuals,  teams, 
crews,  and  units  with  the  learning 
or  performance-aiding 
environment;  automated 
techniques  for  creating  and 
assessing  the  shared  mental 
models  of  crews,  teams,  and 
units. 

Instruction  Delivery 
and  Management: 

Collaborative  team, 
crew,  and  unit 
learning 

Ability  to  foster  optimal  learner-to- 
learner  collaboration  and  learner- 
to-instructor  interaction. 

Comprehensive  representation  of 
collaboration  and  interaction  in 
teams,  crews,  and  units;  principles 
for  establishing  and  supporting 
collaboration  in  distributed 
learning  environments; 
capabilities  to  support  in-route 
mission  rehearsal  despite  physical 
separation  of  participants. 
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Table  3.  Capabilities,  Goals,  and  R&D  Requirements  for  the  Development  of 
Technology  in  Education,  Training,  and  Performance-Aiding  (Continued) 


Capability 

Description/Goal 

Requirements/Gaps 

Instruction  Delivery 
and  Management: 

Self-management 
of  instructional 
progress 

Trainable  techniques  for 
developing  meta-cognitive 
capabilities;  comprehensive  tools 
for  supporting  learner-directed 
instruction  and  learning-to-learn. 

Comprehensive  understanding  of 
meta-cognition  and  its 
development;  comprehensive 
understanding  of  incentive 
management  for  distributed 
learning;  technology-based  tools 
that  allow  distributed  learners  to 
manage  their  progress. 

Instruction  Delivery 
and  Management: 

Security 

maintenance 

Multiple  levels  of  security  provided 
reliably  and  as  needed  across  a 
network  of  distributed  instruction 
and  performance-aiding  based  on 
a  pervasive  security  policy  and 
technical  strategy  that  addresses 
pan-network,  pan-organizational, 
authentication,  and  data  access 
and  security. 

Reliable  learner/user 
identification,  authentication,  and 
authorization;  standard  format  for 
acquisition,  storage, 
maintenance,  retrieval,  and 
application  of  knowledge  bases; 
ability  to  access  both  secure  and 
nonsecure  sources  transparently 
within  the  same  network  for 
training  and  performance-aiding 
purposes. 

Instruction  Delivery 
and  Management: 

Integration  of 
approaches 

Integration  of  continuous, 
asynchronous  instruction  with 
current  classroom,  instructor-led 
approaches. 

Principles  for  combining 
technology-based  with  human- 
based  instructional  resources  and 
approaches;  development  of 
effective  organizational 
structures. 

Instruction  Delivery 
and  Management: 

Instructional 

management 

Ability  to  optimize  student 
progress  toward  achieving 
instructional  or  performance  goals. 

Ability  to  match  instructional  or 
problem-solving  goals  with  current 
state  of  the  student  or  user  and 
generate  or  select  optimal  tutorial 
and/or  problem-solving  strategies. 

Evaluation 

Instruction 

Evaluation: 

Cost  and 

effectiveness 

analyses 

Fully  developed  system  for  cost- 
effectiveness  assessment  of 
education,  training,  and 
performance  aids  at  all  levels  to 
include  user  satisfaction, 
achievement  of  objectives, 
improvement  in  performance  at 
duty  stations,  and  enhancement 
of  readiness  and  effectiveness. 

Comprehensive  models  for  both 
costs  and  effectiveness  of 
education,  training,  and 
performance-aiding;  techniques 
for  linking  education  and  training 
to  duty  station  performance; 
verified  and  valid  capabilities  to 
assess  the  military  value  of 
education,  training,  and 
performance-aiding  in  enhancing 
military  readiness  and 
effectiveness. 
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2.  How  Should  We  Get  From  Here  To  There?  A  Summary  in  a  NATO/PfP 
Research  and  Technology  Context 

Table  3  suggests  a  great  many  specific  efforts  that  could— and,  in  fact,  may— be 
undertaken  cooperatively  by  the  NATO/PfP  community.  In  the  space  available  here,  we  can 
only  refer  the  reader  to  Table  3  as  a  list  of  possibilities.  Nonetheless,  two  broad  areas  that 
offer  specific  and  tangible  opportunities  deserve  mention.  They  concern  the  sharing  of 
information  through  a  commonly  accessible  database  and  the  development  of  guidelines  and 
standards  for  technology-based  education,  training,  and  performance-aiding. 

a.  NATO/PfP  Database 

Some  of  the  efforts  suggested  in  Table  3  may  already  be  underway  and  are  being 
pursued  separately  and  independently  by  different  countries.  Even  though  these  efforts  may 
be  independent,  much  can  be  gained  by  sharing  experiences  and  findings  across  all  coun¬ 
tries  that  have  a  stake  in  their  outcomes. 

Creating  a  NATO/PfP  directory  of  databases  would  be  a  good  first  step  (Fletcher, 
Alluisi,  and  Chatelier,  1993).  This  database  would  point  to  data  elements  that  can  be  found 
in  relevant  databases  elsewhere  and  include  with  information  about  how  each  can  be 
accessed.  Users  would  use  an  on-line  inquiry  to  focus  searches  on  the  specific  information 
required  and  the  identification  of  which  (if  any)  existing  databases  might  have  the  informa¬ 
tion  for  which  they  are  looking.  Retrieving  the  information  would  remain  in  users’  hands. 

The  NATO  directory  would  identify  what  data  elements  are  available  and  how  these 
data  elements  might  be  accessed,  but  the  next  steps  taken  would  be  up  to  the  users.  This 
approach  would  describe  access  to  all  data  elements  in  all  accessible  databases,  the  data  in 
the  data  elements  would  be  kept  up  to  date  by  those  who  “own”  the  database,  and  the 
expenses  for  setup  and  maintenance  would  be  minor.  This  approach  is  not  the  most  efficient 
that  could  be  imagined,  but  it  represents  a  beginning.  Also,  because  of  its  low  cost  and  high 
feasibility,  it  might  increase  communication  and  the  demand  for  more  efficiently  available 
information. 

b.  NATO/PfP  Common  Practices 

The  second  opportunity  is  the  development  and  the  establishment  of  common 
practices  (if  not  standards)  for  developing  sharable  objects  and  guidelines  for  their  use  in 
instructional  systems.  The  objects  would  be  intended  to  be  accessible  on  the  global 
information  grid  (today,  the  World  Wide  Web)  and  used  in  devising  military  education, 
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training,  and  performance-aiding  presentations.  This  work  has  already  been  pursued  by 
organizations  in  the  United  States,  Europe,  and  the  Pacific  Rim.  Much  of  it  has  coalesced 
under  the  DoD’s  Advanced  Distributed  Learning  Initiative ,  which  has  been  developing  the 
Sharable  Courseware  Objects  Reference  Model  (SCORM)  (www.adlnet.org).  Much, 
however,  remains  to  be  done. 

Currently,  efforts  have  focused  on  the  software  engineering  needed  to  create  objects 
that  are  accessible,  durable,  portable,  and  reusable.  Not  only  have  these  efforts  been  suc¬ 
cessful,  but  their  prescriptions  are  also  likely  to  last.  These  objects  have  been  developed  with 
the  direct  support  of  business  and  industry  sectors  who  are  most  likely  to  be  affected  by 
them  but  who  were  actively  seeking  a  catalyst  (in  this  case  the  DoD)  to  act  as  a  honest 
broker  in  organizing,  extracting,  and  documenting  a  set  of  compromises  that  would  result  in 
the  standard  practices  they  needed  if  the  market  for  their  software  tools  were  to  grow.  This 
work  produced  the  SCORM  model. 

How  are  these  objects  likely  to  be  used  in  real-world  applications  now  that  they  are 
becoming  available?  Guidelines  for  purchase  agents,  designers,  implementers,  evaluators, 
and  even  classroom  instructors  are  needed.  These  guidelines  must  be  produced  and  are  just 
now  emerging.  To  some  extent,  the  guidelines  will  respond  to  the  unique  needs  of  those 
who  develop  them.  However,  to  another,  much  larger  extent,  standardization  in  these 
guidelines  will  improve  efficiency,  reduce  costs,  and  enhance  effectiveness  as  much  for 
military  organizations  as  they  will  for  the  market  sought  by  business  and  industry.  Such 
standardization,  in  practice,  seems  a  natural  and  achievable  goal  for  the  NATO/PfP  com¬ 
munity. 

E.  A  FINAL  WORD 

Human  competence  is  an  essential  part  of  every  military  system  and  every  military 
operation.  Its  presence  will  not  guarantee  the  full  performance  expected  from  every  military 
system  or  success  in  every  military  operation,  but  its  absence  is  an  often-noted  cause  of 
failure. 

Few  activities  are  as  important  in  ensuring  the  availability  of  human  competence 
when  and  where  it  is  needed  as  are  education,  training,  and  performance-aiding.  The  appli¬ 
cation  of  human-centered  technology,  with  its  promise  for  reducing  costs  and  increasing 
operational  effectiveness,  seems  to  be  an  economic  and  a  military  imperative.  The  NATO/ 
PfP  research  and  technology  community  has  the  opportunity  and  an  obligation  to  lead  its 
members  into  the  future  promised  by  these  technological  applications. 
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GLOSSARY 


AFB 

Air  Force  Base 

AI 

artificial  intelligence 

APG 

Airplane  General 

C2 

command  and  control 

CAI 

computer-aided  instruction 

CBI 

computer-based  instruction 

DoD 

Department  of  Defense 

DTIC 

Defense  Technical  Information  Center 

ICAI 

intelligent  computer-aided  instruction 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 
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